Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[wptrunner] Support testdriver.js in {ref,print-ref,crash}tests #48486

Merged

Conversation

jonathan-j-lee
Copy link
Contributor

@jonathan-j-lee jonathan-j-lee commented Oct 5, 2024

Implements the wpt {manifest,run} changes proposed in web-platform-tests/rfcs#211.

@jonathan-j-lee jonathan-j-lee force-pushed the wptrunner/testdriver-reftest branch 5 times, most recently from 41b2fb1 to c4a6d69 Compare October 5, 2024 07:25
@jonathan-j-lee jonathan-j-lee reopened this Oct 5, 2024
@jonathan-j-lee jonathan-j-lee reopened this Oct 5, 2024
@jonathan-j-lee jonathan-j-lee force-pushed the wptrunner/testdriver-reftest branch 3 times, most recently from 7b80717 to 25d7265 Compare October 7, 2024 17:22
@jonathan-j-lee jonathan-j-lee force-pushed the wptrunner/testdriver-reftest branch 2 times, most recently from a0230ef to 0efaa69 Compare December 10, 2024 21:55
@jonathan-j-lee jonathan-j-lee marked this pull request as ready for review December 10, 2024 22:10
@wpt-pr-bot wpt-pr-bot added docs infra manifest wptrunner The automated test runner, commonly called through ./wpt run labels Dec 10, 2024
@jonathan-j-lee jonathan-j-lee removed the request for review from sideshowbarker December 10, 2024 22:10
@jonathan-j-lee
Copy link
Contributor Author

PTAL @WeizhongX @jgraham. Chrome and Safari pass the new infrastructure/ tests.

@sideshowbarker sideshowbarker removed their assignment Dec 11, 2024
jonathan-j-lee added a commit that referenced this pull request Dec 13, 2024
Relax the `TESTDRIVER-IN-UNSUPPORTED-TYPE` rule now that
#48486 has landed. 

`wpt lint` still bans `testdriver.js` in manual tests, since there's
no external WebDriver program to drive testdriver.
@mfreed7
Copy link
Contributor

mfreed7 commented Dec 13, 2024

...so is it now ok to move some tests using this capability over to the repo?

@jonathan-j-lee
Copy link
Contributor Author

...so is it now ok to move some tests using this capability over to the repo?

From pure WPT PRs, yes. From chromium/src and @chromium-wpt-export-bot, almost, because the PRs need to be rolled downstream first (https://crrev.com/c/6088124). That CL should land soon, barring CQ dry-run showing unexpected regressions, but the wpt.fyi results with/without this PR suggest that's unlikely.

@jonathan-j-lee
Copy link
Contributor Author

@mfreed7 wpt lint has been updated downstream, so you should be able to upstream your tests now without being blocked by chromium_presubmit. (I've confirmed this in https://crrev.com/c/6088404.)

@mfreed7
Copy link
Contributor

mfreed7 commented Dec 13, 2024

Nice! Thanks for getting this landed.

{I'll give it a try myself, here: https://crrev.com/c/6077774 }

chromium-wpt-export-bot pushed a commit that referenced this pull request Dec 13, 2024
See conversations at:

web-platform-tests/rfcs#211
#13183

As of today, this can land:

#48486

See, e.g.:

https://chromium-review.googlesource.com/c/chromium/src/+/6088404

Change-Id: I43ffb77f93f4fa28e7aaa65beea8780f52f87242
chromium-wpt-export-bot pushed a commit that referenced this pull request Dec 14, 2024
As of #48486, the
WebDriver-based testdriver infrastructure now supports {ref,crash}tests.
Revert most of https://crrev.com/c/5972758 to move the tests back. In
Chromium CQ, moving the tests out of `wpt_internal/` also has the effect
of running the tests with `headless_shell` + `chromedriver`, which
passes these tests.

Bug: None
Change-Id: I396053ab5a7ab2a27558fa1de71af614107a8f02
aarongable pushed a commit to chromium/chromium that referenced this pull request Dec 14, 2024
As of web-platform-tests/wpt#48486, the
WebDriver-based testdriver infrastructure now supports {ref,crash}tests.
Revert most of https://crrev.com/c/5972758 to move the tests back. In
Chromium CQ, moving the tests out of `wpt_internal/` also has the effect
of running the tests with `headless_shell` + `chromedriver`, which
passes these tests.

Bug: None
Change-Id: I396053ab5a7ab2a27558fa1de71af614107a8f02
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/6088404
Reviewed-by: Mike Wasserman <[email protected]>
Commit-Queue: Mike Wasserman <[email protected]>
Auto-Submit: Jonathan Lee <[email protected]>
Cr-Commit-Position: refs/heads/main@{#1396271}
chromium-wpt-export-bot pushed a commit that referenced this pull request Dec 14, 2024
As of #48486, the
WebDriver-based testdriver infrastructure now supports {ref,crash}tests.
Revert most of https://crrev.com/c/5972758 to move the tests back. In
Chromium CQ, moving the tests out of `wpt_internal/` also has the effect
of running the tests with `headless_shell` + `chromedriver`, which
passes these tests.

Bug: None
Change-Id: I396053ab5a7ab2a27558fa1de71af614107a8f02
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/6088404
Reviewed-by: Mike Wasserman <[email protected]>
Commit-Queue: Mike Wasserman <[email protected]>
Auto-Submit: Jonathan Lee <[email protected]>
Cr-Commit-Position: refs/heads/main@{#1396271}
chromium-wpt-export-bot pushed a commit that referenced this pull request Dec 14, 2024
See conversations at:

web-platform-tests/rfcs#211
#13183

As of today, this can land:

#48486

See, e.g.:

https://chromium-review.googlesource.com/c/chromium/src/+/6088404

Change-Id: I43ffb77f93f4fa28e7aaa65beea8780f52f87242
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/6077774
Reviewed-by: Joey Arhar <[email protected]>
Auto-Submit: Mason Freed <[email protected]>
Commit-Queue: Mason Freed <[email protected]>
Cr-Commit-Position: refs/heads/main@{#1396273}
aarongable pushed a commit to chromium/chromium that referenced this pull request Dec 14, 2024
See conversations at:

web-platform-tests/rfcs#211
web-platform-tests/wpt#13183

As of today, this can land:

web-platform-tests/wpt#48486

See, e.g.:

https://chromium-review.googlesource.com/c/chromium/src/+/6088404

Change-Id: I43ffb77f93f4fa28e7aaa65beea8780f52f87242
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/6077774
Reviewed-by: Joey Arhar <[email protected]>
Auto-Submit: Mason Freed <[email protected]>
Commit-Queue: Mason Freed <[email protected]>
Cr-Commit-Position: refs/heads/main@{#1396273}
chromium-wpt-export-bot pushed a commit that referenced this pull request Dec 14, 2024
As of #48486, the
WebDriver-based testdriver infrastructure now supports {ref,crash}tests.
Revert most of https://crrev.com/c/5972758 to move the tests back. In
Chromium CQ, moving the tests out of `wpt_internal/` also has the effect
of running the tests with `headless_shell` + `chromedriver`, which
passes these tests.

Bug: None
Change-Id: I396053ab5a7ab2a27558fa1de71af614107a8f02
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/6088404
Reviewed-by: Mike Wasserman <[email protected]>
Commit-Queue: Mike Wasserman <[email protected]>
Auto-Submit: Jonathan Lee <[email protected]>
Cr-Commit-Position: refs/heads/main@{#1396271}
chromium-wpt-export-bot pushed a commit that referenced this pull request Dec 14, 2024
See conversations at:

web-platform-tests/rfcs#211
#13183

As of today, this can land:

#48486

See, e.g.:

https://chromium-review.googlesource.com/c/chromium/src/+/6088404

Change-Id: I43ffb77f93f4fa28e7aaa65beea8780f52f87242
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/6077774
Reviewed-by: Joey Arhar <[email protected]>
Auto-Submit: Mason Freed <[email protected]>
Commit-Queue: Mason Freed <[email protected]>
Cr-Commit-Position: refs/heads/main@{#1396273}
moz-v2v-gh pushed a commit to mozilla/gecko-dev that referenced this pull request Dec 18, 2024
…sts that use testdriver, a=testonly

Automatic update from web-platform-tests
[wpt][fullscreen] Upstream {ref,crash}tests that use testdriver

As of web-platform-tests/wpt#48486, the
WebDriver-based testdriver infrastructure now supports {ref,crash}tests.
Revert most of https://crrev.com/c/5972758 to move the tests back. In
Chromium CQ, moving the tests out of `wpt_internal/` also has the effect
of running the tests with `headless_shell` + `chromedriver`, which
passes these tests.

Bug: None
Change-Id: I396053ab5a7ab2a27558fa1de71af614107a8f02
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/6088404
Reviewed-by: Mike Wasserman <[email protected]>
Commit-Queue: Mike Wasserman <[email protected]>
Auto-Submit: Jonathan Lee <[email protected]>
Cr-Commit-Position: refs/heads/main@{#1396271}

--

wpt-commits: 70e5fd98c0eed8d024c7743fb476386a36c8b8d0
wpt-pr: 49690
moz-v2v-gh pushed a commit to mozilla/gecko-dev that referenced this pull request Dec 18, 2024
… tests out of wpt_internal, a=testonly

Automatic update from web-platform-tests
Move all customizable-<select> reference tests out of wpt_internal

See conversations at:

web-platform-tests/rfcs#211
web-platform-tests/wpt#13183

As of today, this can land:

web-platform-tests/wpt#48486

See, e.g.:

https://chromium-review.googlesource.com/c/chromium/src/+/6088404

Change-Id: I43ffb77f93f4fa28e7aaa65beea8780f52f87242
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/6077774
Reviewed-by: Joey Arhar <[email protected]>
Auto-Submit: Mason Freed <[email protected]>
Commit-Queue: Mason Freed <[email protected]>
Cr-Commit-Position: refs/heads/main@{#1396273}

--

wpt-commits: 2134b52a0ed40aeb768cd2c2856db9b4724527a7
wpt-pr: 49688
ErichDonGubler pushed a commit to erichdongubler-mozilla/firefox that referenced this pull request Dec 19, 2024
…sts that use testdriver, a=testonly

Automatic update from web-platform-tests
[wpt][fullscreen] Upstream {ref,crash}tests that use testdriver

As of web-platform-tests/wpt#48486, the
WebDriver-based testdriver infrastructure now supports {ref,crash}tests.
Revert most of https://crrev.com/c/5972758 to move the tests back. In
Chromium CQ, moving the tests out of `wpt_internal/` also has the effect
of running the tests with `headless_shell` + `chromedriver`, which
passes these tests.

Bug: None
Change-Id: I396053ab5a7ab2a27558fa1de71af614107a8f02
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/6088404
Reviewed-by: Mike Wasserman <[email protected]>
Commit-Queue: Mike Wasserman <[email protected]>
Auto-Submit: Jonathan Lee <[email protected]>
Cr-Commit-Position: refs/heads/main@{#1396271}

--

wpt-commits: 70e5fd98c0eed8d024c7743fb476386a36c8b8d0
wpt-pr: 49690
ErichDonGubler pushed a commit to erichdongubler-mozilla/firefox that referenced this pull request Dec 19, 2024
… tests out of wpt_internal, a=testonly

Automatic update from web-platform-tests
Move all customizable-<select> reference tests out of wpt_internal

See conversations at:

web-platform-tests/rfcs#211
web-platform-tests/wpt#13183

As of today, this can land:

web-platform-tests/wpt#48486

See, e.g.:

https://chromium-review.googlesource.com/c/chromium/src/+/6088404

Change-Id: I43ffb77f93f4fa28e7aaa65beea8780f52f87242
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/6077774
Reviewed-by: Joey Arhar <[email protected]>
Auto-Submit: Mason Freed <[email protected]>
Commit-Queue: Mason Freed <[email protected]>
Cr-Commit-Position: refs/heads/main@{#1396273}

--

wpt-commits: 2134b52a0ed40aeb768cd2c2856db9b4724527a7
wpt-pr: 49688
gecko-dev-updater pushed a commit to marco-c/gecko-dev-wordified-and-comments-removed that referenced this pull request Jan 1, 2025
…sts that use testdriver, a=testonly

Automatic update from web-platform-tests
[wpt][fullscreen] Upstream {ref,crash}tests that use testdriver

As of web-platform-tests/wpt#48486, the
WebDriver-based testdriver infrastructure now supports {ref,crash}tests.
Revert most of https://crrev.com/c/5972758 to move the tests back. In
Chromium CQ, moving the tests out of `wpt_internal/` also has the effect
of running the tests with `headless_shell` + `chromedriver`, which
passes these tests.

Bug: None
Change-Id: I396053ab5a7ab2a27558fa1de71af614107a8f02
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/6088404
Reviewed-by: Mike Wasserman <mswchromium.org>
Commit-Queue: Mike Wasserman <mswchromium.org>
Auto-Submit: Jonathan Lee <jonathanjleegoogle.com>
Cr-Commit-Position: refs/heads/main{#1396271}

--

wpt-commits: 70e5fd98c0eed8d024c7743fb476386a36c8b8d0
wpt-pr: 49690

UltraBlame original commit: d40b2c8ff387c21e9289b92afdc86dcad1e4c426
gecko-dev-updater pushed a commit to marco-c/gecko-dev-wordified-and-comments-removed that referenced this pull request Jan 1, 2025
… tests out of wpt_internal, a=testonly

Automatic update from web-platform-tests
Move all customizable-<select> reference tests out of wpt_internal

See conversations at:

web-platform-tests/rfcs#211
web-platform-tests/wpt#13183

As of today, this can land:

web-platform-tests/wpt#48486

See, e.g.:

https://chromium-review.googlesource.com/c/chromium/src/+/6088404

Change-Id: I43ffb77f93f4fa28e7aaa65beea8780f52f87242
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/6077774
Reviewed-by: Joey Arhar <jarharchromium.org>
Auto-Submit: Mason Freed <masonfchromium.org>
Commit-Queue: Mason Freed <masonfchromium.org>
Cr-Commit-Position: refs/heads/main{#1396273}

--

wpt-commits: 2134b52a0ed40aeb768cd2c2856db9b4724527a7
wpt-pr: 49688

UltraBlame original commit: 31380657f025c9e677795d2107c4264f45cfc0a6
gecko-dev-updater pushed a commit to marco-c/gecko-dev-comments-removed that referenced this pull request Jan 1, 2025
…sts that use testdriver, a=testonly

Automatic update from web-platform-tests
[wpt][fullscreen] Upstream {ref,crash}tests that use testdriver

As of web-platform-tests/wpt#48486, the
WebDriver-based testdriver infrastructure now supports {ref,crash}tests.
Revert most of https://crrev.com/c/5972758 to move the tests back. In
Chromium CQ, moving the tests out of `wpt_internal/` also has the effect
of running the tests with `headless_shell` + `chromedriver`, which
passes these tests.

Bug: None
Change-Id: I396053ab5a7ab2a27558fa1de71af614107a8f02
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/6088404
Reviewed-by: Mike Wasserman <mswchromium.org>
Commit-Queue: Mike Wasserman <mswchromium.org>
Auto-Submit: Jonathan Lee <jonathanjleegoogle.com>
Cr-Commit-Position: refs/heads/main{#1396271}

--

wpt-commits: 70e5fd98c0eed8d024c7743fb476386a36c8b8d0
wpt-pr: 49690

UltraBlame original commit: d40b2c8ff387c21e9289b92afdc86dcad1e4c426
gecko-dev-updater pushed a commit to marco-c/gecko-dev-comments-removed that referenced this pull request Jan 1, 2025
… tests out of wpt_internal, a=testonly

Automatic update from web-platform-tests
Move all customizable-<select> reference tests out of wpt_internal

See conversations at:

web-platform-tests/rfcs#211
web-platform-tests/wpt#13183

As of today, this can land:

web-platform-tests/wpt#48486

See, e.g.:

https://chromium-review.googlesource.com/c/chromium/src/+/6088404

Change-Id: I43ffb77f93f4fa28e7aaa65beea8780f52f87242
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/6077774
Reviewed-by: Joey Arhar <jarharchromium.org>
Auto-Submit: Mason Freed <masonfchromium.org>
Commit-Queue: Mason Freed <masonfchromium.org>
Cr-Commit-Position: refs/heads/main{#1396273}

--

wpt-commits: 2134b52a0ed40aeb768cd2c2856db9b4724527a7
wpt-pr: 49688

UltraBlame original commit: 31380657f025c9e677795d2107c4264f45cfc0a6
gecko-dev-updater pushed a commit to marco-c/gecko-dev-wordified that referenced this pull request Jan 1, 2025
…sts that use testdriver, a=testonly

Automatic update from web-platform-tests
[wpt][fullscreen] Upstream {ref,crash}tests that use testdriver

As of web-platform-tests/wpt#48486, the
WebDriver-based testdriver infrastructure now supports {ref,crash}tests.
Revert most of https://crrev.com/c/5972758 to move the tests back. In
Chromium CQ, moving the tests out of `wpt_internal/` also has the effect
of running the tests with `headless_shell` + `chromedriver`, which
passes these tests.

Bug: None
Change-Id: I396053ab5a7ab2a27558fa1de71af614107a8f02
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/6088404
Reviewed-by: Mike Wasserman <mswchromium.org>
Commit-Queue: Mike Wasserman <mswchromium.org>
Auto-Submit: Jonathan Lee <jonathanjleegoogle.com>
Cr-Commit-Position: refs/heads/main{#1396271}

--

wpt-commits: 70e5fd98c0eed8d024c7743fb476386a36c8b8d0
wpt-pr: 49690

UltraBlame original commit: d40b2c8ff387c21e9289b92afdc86dcad1e4c426
gecko-dev-updater pushed a commit to marco-c/gecko-dev-wordified that referenced this pull request Jan 1, 2025
… tests out of wpt_internal, a=testonly

Automatic update from web-platform-tests
Move all customizable-<select> reference tests out of wpt_internal

See conversations at:

web-platform-tests/rfcs#211
web-platform-tests/wpt#13183

As of today, this can land:

web-platform-tests/wpt#48486

See, e.g.:

https://chromium-review.googlesource.com/c/chromium/src/+/6088404

Change-Id: I43ffb77f93f4fa28e7aaa65beea8780f52f87242
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/6077774
Reviewed-by: Joey Arhar <jarharchromium.org>
Auto-Submit: Mason Freed <masonfchromium.org>
Commit-Queue: Mason Freed <masonfchromium.org>
Cr-Commit-Position: refs/heads/main{#1396273}

--

wpt-commits: 2134b52a0ed40aeb768cd2c2856db9b4724527a7
wpt-pr: 49688

UltraBlame original commit: 31380657f025c9e677795d2107c4264f45cfc0a6
i3roly pushed a commit to i3roly/firefox-dynasty that referenced this pull request Jan 2, 2025
…sts that use testdriver, a=testonly

Automatic update from web-platform-tests
[wpt][fullscreen] Upstream {ref,crash}tests that use testdriver

As of web-platform-tests/wpt#48486, the
WebDriver-based testdriver infrastructure now supports {ref,crash}tests.
Revert most of https://crrev.com/c/5972758 to move the tests back. In
Chromium CQ, moving the tests out of `wpt_internal/` also has the effect
of running the tests with `headless_shell` + `chromedriver`, which
passes these tests.

Bug: None
Change-Id: I396053ab5a7ab2a27558fa1de71af614107a8f02
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/6088404
Reviewed-by: Mike Wasserman <[email protected]>
Commit-Queue: Mike Wasserman <[email protected]>
Auto-Submit: Jonathan Lee <[email protected]>
Cr-Commit-Position: refs/heads/main@{#1396271}

--

wpt-commits: 70e5fd98c0eed8d024c7743fb476386a36c8b8d0
wpt-pr: 49690
i3roly pushed a commit to i3roly/firefox-dynasty that referenced this pull request Jan 2, 2025
… tests out of wpt_internal, a=testonly

Automatic update from web-platform-tests
Move all customizable-<select> reference tests out of wpt_internal

See conversations at:

web-platform-tests/rfcs#211
web-platform-tests/wpt#13183

As of today, this can land:

web-platform-tests/wpt#48486

See, e.g.:

https://chromium-review.googlesource.com/c/chromium/src/+/6088404

Change-Id: I43ffb77f93f4fa28e7aaa65beea8780f52f87242
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/6077774
Reviewed-by: Joey Arhar <[email protected]>
Auto-Submit: Mason Freed <[email protected]>
Commit-Queue: Mason Freed <[email protected]>
Cr-Commit-Position: refs/heads/main@{#1396273}

--

wpt-commits: 2134b52a0ed40aeb768cd2c2856db9b4724527a7
wpt-pr: 49688
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
docs infra manifest wptrunner The automated test runner, commonly called through ./wpt run
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants